WIP: Holistic CLI test harness experiments #98

epage · 2018-03-27T23:14:09Z

This includes

Experiments with extension traits, opening the door for duct, see Use duct internally? #32
API for quickly setting up a file system for testing, mostly inspired by my work on cobalt ("I'd like to test X but that'll be too much work to set up a fixture. Maybe its good enough...").
Comparisons with cli_test_dir, another holistic harness that is more rigid in its API

See commit messages for more details

Work to do

Replace Asserts ability to run commands with traits
Make it easy to setup a tempdir
Add assertion predicates
Determine crate(s) structure
Tests and documentation

azriel91 · 2018-03-28T03:56:44Z

btw, tempdir is now merged into tempfile 3.0.0 and was published 2 days ago.

This is an experiment in trying to use extension traits rather than wrapping `process::Command`. This both makes it more extensible (can interop with other crates) and able to be adapted to other "Command" crates like `duct`. `cli_test_dir` has something like `CommandStdInExt` called `CommandExt`. Differences include: - Scoped name since traits generally are pulled out of any namespace they are in. - Preserves the command and `stdin` to for richer error reporting.

This is an experiment in what kind of tempdir operations a holistic CLI testing framework might provide, following on the previous experiments with extension traits. The exact structure in this crate or across crates is TBD. This crate extends `TempDir` with the following - In TempDir or a child path, run a command. - On child path, touch a file. - On child path, write a binary blob or str to file. - Copy to a TempDir or a child path some files. Some other potential operations include - `write_yml(serde)` - `write_json(serde)` - `write_toml(serde)` In contrast, `cli_test_dir` can: - Run a single pre-defined program within the tempdir - Write binary files to tempdir - Offer a absolute path to a child file within the crate source (so its safe to pass to the program running in the tempdir).

Another common use case in basic assertions.

killercup · 2018-04-03T19:24:54Z

fyi @nikomatsakis just told me he used assert_cli and manually added a tempdir wrapper because that was missing

peddermaster2 · 2018-04-03T19:59:11Z

src/cmd.rs

+
+impl CommandCargoExt for process::Command {
+    fn main_binary() -> Self {
+        let mut cmd = process::Command::new("carg");


carg -> cargo?
Same in line 51

Thanks!

Obviously very little testing has been done which might be clear considering this is here :). This is a rough sketch to get high level feedback before actually getting the details right.

epage · 2018-05-08T19:05:38Z

My current thought on crate organization

assert_cmd for writing asserts on std::process::Command
assert_fs for tempfile / path assertions
assert_cli that re-exports both of the above

Couple questions from that

I assume some people would appreciate process::Command helpers that this PR provides but aren't interested in assertions. Should we advertise that we provide these or create yet another crate for them?
- passing in a Vec<u8> for stdin
- Turning an Command / Output into a Result
Currently, the tempfile parts have a distinct struct for playing with files in the tempdir. The cmd parts then support using that to run a command in that directory. This might run afoul of trait rules if we split this into multiple crates.
- Will it?
- If it will, should we just switch to attaching these extension traits to Path instead?

killercup

Very cool! Sorry it took me so long to around to this, @epage.

I've long been in favor of splitting assert_cli up into smaller pieces. What we should also offer in the long run, though, is a way to easily extend this, e.g., to make shortcut methods to set a bunch of env vars.

killercup · 2018-05-24T11:18:14Z

src/cmd.rs

+    /// ```
+    fn main_binary() -> Self;
+
+    /// Create a `Command` Run a specific binary of the current crate.


s/Run/to run

killercup · 2018-05-24T11:19:09Z

src/cmd.rs

+/// `std::process::Output` represented as a `Result`.
+pub type OutputResult = Result<process::Output, OutputError>;
+
+/// Extends `std::process::Output` with methods to to convert it to an `OutputResult`.


killercup · 2018-05-24T11:21:53Z

src/cmd.rs

+    ///
+    /// By default, stdin, stdout and stderr are inherited from the parent.
+    ///
+    /// *(mirrors `std::process::Command::spawn`**


killercup · 2018-05-24T11:33:41Z

src/temp.rs

+
+/// Extend `TempDir` to run commands in it.
+pub trait TempDirCommandExt {
+    /// Constructs a new Command for launching the program at path program, with the following


s/path program/path

killercup · 2018-05-24T11:35:19Z

src/temp.rs

+    /// temp.command("pwd").output().unwrap();
+    /// temp.close().unwrap();
+    /// ```
+    fn command<S>(&self, program: S) -> process::Command


i'd probably call this exec?

epage · 2018-05-29T12:30:43Z

This is now implemented as

epage added 2 commits March 29, 2018 10:37

epage force-pushed the trait branch from a211010 to 9767301 Compare March 29, 2018 16:38

epage added 2 commits March 29, 2018 11:35

feat(cmd): Add unwrap_err to commands

8b0f56c

Another common use case in basic assertions.

feat(cmd): First pass at assertions

300ece0

killercup mentioned this pull request Apr 3, 2018

CLI WG Newsletter #2 rust-cli/team#19

Closed

peddermaster2 reviewed Apr 3, 2018

View reviewed changes

vincentdephily mentioned this pull request Apr 5, 2018

Allow checking for any command exit value. #99

Closed

This was referenced Apr 7, 2018

Regex matching. #85

Closed

Feature: Or-Operator #19

Closed

Multiple output matchers #20

Closed

Fluent Output Tests Look off with rustfmt #70

Closed

Regex matching in OutputAssertionBuilder #81

Closed

Add OutputAssertionBuilder::empty() #82

Closed

epage mentioned this pull request May 15, 2018

Feature Request: wait and/or then methods after writing to stdin #106

Closed

killercup approved these changes May 24, 2018

View reviewed changes

volks73 mentioned this pull request May 24, 2018

Add optional access to Command's stdin handle #107

Open

4 tasks

epage closed this May 29, 2018

epage deleted the trait branch May 29, 2018 12:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Holistic CLI test harness experiments #98

WIP: Holistic CLI test harness experiments #98

epage commented Mar 27, 2018

azriel91 commented Mar 28, 2018

killercup commented Apr 3, 2018

peddermaster2 Apr 3, 2018

epage Apr 3, 2018

epage commented May 8, 2018

killercup left a comment

killercup May 24, 2018

killercup May 24, 2018

killercup May 24, 2018

killercup May 24, 2018

killercup May 24, 2018

epage commented May 29, 2018

WIP: Holistic CLI test harness experiments #98

WIP: Holistic CLI test harness experiments #98

Conversation

epage commented Mar 27, 2018

azriel91 commented Mar 28, 2018

killercup commented Apr 3, 2018

peddermaster2 Apr 3, 2018

Choose a reason for hiding this comment

epage Apr 3, 2018

Choose a reason for hiding this comment

epage commented May 8, 2018

killercup left a comment

Choose a reason for hiding this comment

killercup May 24, 2018

Choose a reason for hiding this comment

killercup May 24, 2018

Choose a reason for hiding this comment

killercup May 24, 2018

Choose a reason for hiding this comment

killercup May 24, 2018

Choose a reason for hiding this comment

killercup May 24, 2018

Choose a reason for hiding this comment

epage commented May 29, 2018